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DEVELOPING  SOFTWARE  FOR  EASE  OF  CHANGE;  METRICS  FROM 
LATER  IN  THE  TRALAB  SYSTEM  LIFE  CYCLE 


1.0  Introduction 

This  paper  is  a  summaiy  of  software  change  and  defect  data  collected  during  extended 
development  and  post-deployment  software  suppoit  (PDSS)  for  the  Training  Laboratory 
(TRALAB),  a  computer-bas^  training  system  developed  starting  in  1984  by  the  Naval  Center  for 
Space  Technology  (NCST)  at  the  Naval  Research  Lalraratoiy  (NRL).  Hager  [89]  describes  the 
TRALAB  project  in  detail.  Of  note,  the  development  contractor  was  required  to  apply  software 
engineering  technology  developed  previously  at  NRL  as  part  of  the  Software  Cost  Reduction 
(SCR)  project  [CLEMENTS  86;  HENING^  80;  PARNAS  84].  Another  important  requirement 
was  for  the  collection  of  project  metric  data  that  could  be  used  to  evaluate  the  effectiveness  of  the 
SCR  lyjproach.  Data  collection  was  accomplished  by  modifying  customary  project  Software 
Problem  Reports  (SPRs). 

Analysis  of  the  data  collected  on  SPR  resolution  work  during  extended  development 
integration  and  test  (I&T)  during  August  1988  through  April  1989  indicates  that  the  ^plication  of 
SCR  technology  enhanc^  software  ease  of  change  [HAGER  91].  This  paper  is  a  continuation  of 
the  earlier  analysis  to  include  PDSS  SPR  data  collet^  during  August  1990  through  May  1991. 
The  mote  recent  analysis  continues  to  suggest  that  identifying  expected  system  ch^ges  during 
system  definition  stages  and  modularizing  the  system  to  encapsulate  these  changes  yield  life-cycle 
benefits.  Modifications  required  by  changes  tend  to  be  confined  to  a  small  number  of  design 
components,  and  it  is  easier  to  implement  expected  changes  in  comparison  to  arbitrary  changes. 

The  remainder  of  this  paper  comprises  five  sections.  Section  2  is  a  summary  of  NRL’s 
earlier  SCR  project,  fimdamental  SCR  design  and  documentation  concepts,  and  the  TRALAB 
project  and  architecture.  Section  3  contains  analyses  of  the  SPR  data.  Section  4  provides 
aclmowledgments;  section  5,  references.  Appendix  A  provides  the  Software  Problem  Report 
(SPR)  and  Software  Modification  Transmittal  (SMT)  forms  used  to  collect  and  report  the  TRALAB 
change  and  problem  data. 

2.0  Background 

2.1  Problem 

The  difficulty  of  generating  software  that  is  easily  maintained  is  evident  when  full  software 
life-cycle  costs  are  examined.  Typically,  PDSS  efforts  account  for  60%  or  more  of  the  total  life- 
cycle  costs  [BOEHM  73].  One  reason  for  such  a  situation  may  be  that  ease  of  maintenance  is  not  a 
natural  by-product  of  many  currently  used  development  approaches  and  methodologies.  For 
exan^le: 

-  maintainability  requirements  are  omitted  or  not  clearly  identified  in  system  requirements 
specifications, 

-  verification  of  maintainability  requirements  is  imprecise  at  best, 

-  documentation  structures  do  not  provide  enough  visibility  for  maintainability  concerns, 
and 

-  design  approaches  do  not  address  directly  how  to  ensure  that  modifications  required  by 
eventual  ibcely  changes  will  not  ripple  tlroughout  the  design  and  implementation. 

2.2  Software  Cost  Reduction  Aoptoach 

Controlling  software  maintenance  costs  would  seem  to  require  changes  in  both  software 
design  approaches  [CLEMENTS  86;  LAMB  88;  CLEMENTS  83;  WALLACE  87;  PARNAS  72] 
and  supporting  documentation  structures  [HESTER  81].  In  1978,  the  SCR  program  was  initiated 
by  NRL  and  tte  Naval  Wetqrans  Center  to  develop  and  illustrate  software  technology  that  could  be 
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used  to  control  software  life-cycle  costs.  Analyses  of  early  SCR  project  metrics  showed  patterns 
that  suggest  that  SCR  technology  can  help  to  control  such  costs  [CHMURA  90]. 

A  fundamental  aspect  of  the  SCR  approach  was  the  principle  of  information  hiding,  a 
software  design  concept  first  introduced  by  Pamas  [72].  With  information  hiding,  a  software 
“m^ule”  is  treated  both  as  a  unit  of  work  and  as  a  unit  of  rework.  Accordingly,  software 
designers  should  consider  likely  future  changes  when  identifying  software  monies  to  prevent 
unpr^ctable  and  possibly  wi^-ranging  rework  in  the  future.  first  step  in  effectively 
relying  the  information  Uding  principle  is  to  identify  expected  changes  early  in  the  requirements 
sp^fication  process.  Once  the  expected  changes  are  agr^  upon,  they  are  prioritized  based  on 
their  likeliho<^  of  occurrence.  The  expected  changes  are  then  factored  explicitly  into  the  design  of 
the  system.  The  general  approach  is  to  identify  and  specify  rinxiules  that  limit  the  rework  required 
of  future  changes  by  trying  to  confine  or  “hide"  each  expected  change  in  separate  modules,  which 
are  referred  to  as  information-hiding  modules.  The  information  that  will  be  hidden  behind  an 
“abstract"  programming  interface  to  an  information-hiding  module  is  referred  to  as  the  “secret"  of 
the  rtKxlule.  Tlie  TRALAB  design  architecture  described  in  section  2.4  below  is  an  information¬ 
hiding  module  structure. 

2.3  TRALAB  Program 

The  TRALAB  program  started  in  September  1984  with  the  purpose  of  developing  a 
computer-based  training  system  to  support  several  data  collectitni  and  processing  systems 
developed  and  maintained  by  the  NCST  at  NRL  [HAGER  89].  Following  a  successful  System 
Elefinition  phase,  NRL  redirected  the  effort  to  incorporate  SCR  concerns.  For  example,  a  list  of 
expected  TRALAB  dianges  was  generated  and  refined  over  the  course  of  the  initial  (tevelopment, 
and  an  infomiation-hiding  design  approach  was  used.  The  rationale  for  the  redirection  was  (1)  to 
reduce  the  risks  associated  with  the  history  of  frequent  and  significant  upgrades  experienced  with 
the  NCST  data  collection  and  processing  systems,  and  (2)  to  experiment  with  the  SCR  technology 
in  the  development  of  a  new  NCST  system. 

A  TRALAB  system  comprises  26  Zenith  286  personal  corrqruters,  networked  via  Ethernet. 
System  users  consist  of  Students,  Courseware  Authors,  Instructors,  and  Administrators. 
InstructtHS  have  the  ability  to  interact  and  monitor  students  engaged  in  training  on  TRALAB 
computers. 

A  TRALAB  system  consists  of  approximately  49,000  lines  of  non-conunented  logical  source 
lines  (NCLSLOC)  ^AUMERT  92]  of  Pascal  code,  25,000  database  records,  and  an  additional 
10,000  scenario  script  records.  Logical  source  statements  have  a  defined  beginning  and  ending, 
independent  of  any  relationship  to  the  physical  lines  on  which  it  is  recorded  or  printed. 
Descriptions  of  user-interface  menus  and  forms,  as  well  as  messages  and  displays  characteristic  of 
a  “target  system”  for  which  TRALAB  would  offer  computer-based  training  are  stored  as  database 
entries  to  facilitate  easy  capture  and  change.  Scenarios  ate  script  files  used  to  control  the  flow  of 
the  training  simulations.  Scenarios  ate  created  and  maintained  by  trainers  using  utilities  genetrued 
as  part  of  tiie  development  and,  by  design,  do  not  requite  programming  skills  to  develop  or 
maintain. 

TRALAB  modules  ate  implemented  as  Pascal  packages  (non- ANSI  standard 
inqrlementation).  Pascal  packages  are  similar  to  the  Ada  package  construct  and  provide  support  for 
information  hid^g  concerns  through  separately  compilable  interfaces  (visible  to  a  programmer)  and 
inqrlementations  (hidden). 
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Table  2.3-1  is  a  list  of  the  latest  sizes  of  each  of  the  third-level  modules  in  terms  of 
NCLSLCX!.  The  NCLSLOC  count  for  Simulation  Activity  is  an  aggregate.  The  Simulation 
Activity  module  comprises  three  fourth-level  target-system-specific  mc^ules. 


Table  2J«1  Third-Level  Module  Size 

MODULE 

SOURCE  LINES  OF  CODE  (NCLSLOC) 

HARDWARE-HIDING 

VIRTUAL  NETWORK 

1071 

VIRTUAL  TERMINAL 

2869 

BEHAVIOR-HIDING 

DATA  BASE  EDITOR 

1675 

SYSTEM  ADMINISTRATION 

1740 

SIMULATION  ACTIVITY 

12254 

TRAINING  EVALUATION 

1724 

STUDENT  OBSERVATION 

946 

MENU  PROCESSING 

4599 

CONTROL 

718 

SOFTWARE  DECISION-HIDING 

SCENARIO  TRANSLATION 

2553 

SIMULATION  DELIVERY 

3417 

VIRTUAL  DATABASE 

602 

SYSTEM  DATA 

2622 

TARGET  DATA 

2556 

SYSTEM  ADMINISTRATION  DATA 

1946 

OreRATIONALDATA 

1631 
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The  TRALAB  module  structure  is  hierarchical  where  a  submodule  of  a  module  hides  one  or 
more  of  the  secrets  of  the  parent  module.  The  first  two  levels  of  the  design  hierarchy  provide  an 
organizational  road  rnry)  for  identifying  what  collections  of  software  modules/components  might 
ne^  to  be  riKxIified  in  response  to  a  change.  These  upper  levels  of  the  information-hiding  design 
are  somewhat  generic  and  may  apply  to  many  systems.  Modules  at  level  3  and  below  tend  to  be 
objects  that  wifi  be  implemented  in  code  (e.g.,  as  Pascal  or  ADA  packages).  There  are  four  levels 
in  the  TRALAB  hierarchy,  the  first  level  of  which  consisted  of  the  following: 

-  Hardware-Hiding  module 

-  Behavior-Hiding  module 

-  Software  Decision-Hiding  module 

The  designers  of  the  TRALAB  module  structure  did  not  adjust  the  design  decomposition 
based  on  NQ.,SLOC  size  or  complexity  of  a  module  implementation.  The  decomposition  strategy 
relied  entirely  on  encqrsulating  expected  changes  within  distinct  components  of  tlK  system.  This 
philosophy  was  a  departure  from  rational  development  approaches  that  limit  module  size  and 
complexity  to  support  span  of  control  and  testability  concerns. 

2.4.1  Hardware- Hiding  Module 

The  Hardware-Hiding  nnodule  con:^)rises  modules  that  would  need  to  be  modified  when 
system  or  interface  hardware  is  replaced  or  modified.  The  Hardware-Hiding  nKxlule  includes 
(i.e.,  is  decomposed  into)  the  Extended  Computer  and  the  Device  Interface  mxlules. 

The  Extended  Conq)uter  module  hides  those  characteristics  of  the  computing  platform  that  are 
likely  to  change  if  die  computer  or  its  operating  system  is  modified  or  repla^.  It  offers  a  virtual 
computer  for  the  remaining  TRALAB  software.  The  major  submodule  of  the  Extended  Computer 
module  is  die  Virtual  Oper^g  System  (VOS)  module  that  hides  some  of  the  peculiarities  and 
changetdile  features  of  the  OS  version  used  initially  for  TRALAB.  It  should  be  noted  that  the  VOS 
offers  features  similar  to  the  initial  OS;  in  other  words,  TRALAB  software  engineers  were  careful 
not  to  commit  to  design  a  new  portable  OS.  One  area  where  this  philosophy  influenced  the 
development  qiproach  was  the  TRALAB  executive  module  (TCN).  A  lack  of  tasking  primitives 
in  the  underlying  OS  led  to  a  polling  approach  to  event  notification  and  processing.  No  attempt  was 
made  to  extend  existing  OS  functionality  to  support  real-time  tasking  concerns  by  adding  new 
tasking  primitives. 

The  Device  Interface  module  conpises  two  third-level  submodules  -  the  Virtual  Network 
Intoface  (VNI)  and  the  Virtual  Tominal  ^TM)  modules.  The  VNI  riKxiule  hides  the  commercial 
network  software  used.  The  abstract  interface  specifications  written  for  this  module  identify 
^  abstract  network  primitives  for  estid)lishing  a  circuit,  sending  a  message,  and  receiving  a  message. 
If  the  technology  for  establishing  a  circuit  or  sending  a  messs^  changes,  other  modules  using  tiie 
VNI  service  routines  do  not  have  to  be  modified. 

The  Virtual  Terminal  module  insulates  die  system  fiom  changes  to  the  terminal  by 
providing  abstract  primitives  for  screen  display  and  keyboard  drivers.  The  display  output  device  is 
managed  as  a  set  of  windows,  each  with  characteristics  to  simulate  portions  of  target  screen 
displays.  The  Virtual  Terminal  Interface  hides  the  physical  characteristics  of  the  display  device, 
locations  of  the  devices,  and  windowing  mechanisms.  The  virtual  interface  provides  the  cq)ability 
to  change  physicd  screen  characteristics  without  impacting  existing  software. 
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Figure  2.4. 1-1  illustrates  the  composition  of  the  Hardware-Hiding  Module. 


Figure  2.4.1«1.  Hardware<Hiding  Module  and  Submodules 
2.4.2  Behavior-Hiding  Module 

The  Behavior-Hiding  module  comprises  modules  that  would  need  to  be  modified  if  there 
are  changes  to  the  required  system  behavior  spedfied  in  the  TRALAB  System  Technical 
Specification  [HAGER  89].  The  Behavior-Hiding  module  is  deconqws^  into  two  second-level 
modules:  die  Application  Mver  module  and  the  Shared  Service  module. 

The  Application  Driver  (AD)  module  is  the  sole  controller  of  sets  of  closely  related  outputs. 
Each  of  its  thi^-level  submodules  hides  the  rules  determining  the  values  of  the  outputs  and  the  data 
structures  and  algorithms  necessary  to  implement  the  outputs.  Expected  changes  d^t  with  in  the 
AD  module  include  the  authoring  exchange  necessary  to  create  and  maintain  scenarios,  the  system 
administrator  exchange  necessary  to  maintain  target  system  databases,  the  system  administration 
classroom  management  policies,  the  processing  unique  to  specific  target  system  simulations,  the 
student  evaluation  processing  and  criteria,  and  the  student  monitoring  processing.  A  change  in  one 
of  these  areas  should  be  coni^ed  to  a  specific  third-level  AD  submodule. 

The  Shared  Service  module  comprises  software  that  controls  r^uired  external  behavior 
common  to  two  or  more  AD  modules.  The  Shared  Service  modules  hide  the  characteristics  of  the 
sha^  tehavior  and  tte  algorithms  and  data  structures  necessary  to  implement  the  shared  behavior. 
Some  of  the  secrets  of  the  Shared  Service  module  include  AD  module  initialization,  menu  services, 
and  control  structures  common  to  the  AD  modules.  A  change  in  any  of  these  areas  is  isolated  to 
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the  Shared  Service  modules,  even  though  the  change  may  affect  an  external  behavior  shared  by 
many  {^)plication  modules. 

All  required  TRALAB  system  behavior  is  provided  by  the  Behavior-Hiding  modules. 
During  eariy  design  work,  design  credibility  is  est^lished  by  mapping  the  required  system 
behavior  identified  in  the  System  Requirements  Specification  to  Behavior-Hiding  modules. 

Figure  2.4.2- 1  illustrates  the  composition  of  the  Behavior-Hiding  Module. 


Figure  2.4.2>1.  Behavior-HIding  Module  and  Submodules 


2.4.3  Software  Decision-Hiding  Module 


The  Software  Decision-Hiding  module  conqirises  modules  that  would  need  to  be  modified 
if  there  are  changes  to  designer-generated  decisions.  For  exan^le,  the  choice  of  a  specific 
algorithm  not  specified  in  Ae  System  Requirements  Specification  is  a  designer-generated  decision. 


The  Software  Decision-Hiding  module  is  decomposed  into  three  second-level  modules:  the 
Scenario  Interface  module,  the  Database  Utilities  noodule,  and  the  System  Generation  module. 

The  Scenario  Interface  module  hides  changes  to  the  training  scenario  validation  policies,  the 
translation  process  ^m  dMs  external  scenario  language  used  by  the  authors  to  the  internal  scenario 
primitives,  and  the  execution  of  those  primitives.  All  algorithms  to  parse,  validate,  translate,  and 
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execute  the  scenarios  are  hidden  in  these  modules.  These  changes  were  allocated  to  Software 
Decision-Hiding  nKxiules  because  the  specific  l^guage  implementation  necessary  to  support 
required  training  was  designer-determined. 


The  Database  Utilities  module  consists  of  software  that  needs  to  be  modified  if  changes  are 
ma^  to  the  database  man^ement  system  or  to  the  internal  storage,  retrieval,  or  maintenance 
policies.  To  insulate  implication  modules  from  the  underlying  datiii)ase  management  system,  a 
Virtual  Database  Interface  module  is  provided.  It  provides  the  file  management  primitives 
necessary  to  support  indexed  sequential  access  da^  retrieval.  Any  changes  to  the  data  access 
policies  are  limited  to  this  tiKxlule.  Target  System,  System  Administration,  Operational,  and 
Scenario  Data  modules  provide  the  Application  Programming  Interface  (API)  services  for 
TRALAB  databases.  Knowledge  of  ^  physical  representation  of  the  TRALAB  data  was  hidden 
from  consumer  modules  throu^  these  abstract  interfaces. 

The  System  Generation  module  hides  the  expected  changes  related  to  the  software 
processing  environment  and  the  underlying  language.  It  hides  the  command  structures  necessary 
to  compile  and  link  the  software,  values  of  system  generation  parameters  that  select  different 
implementations  of  a  module,  and  specialized  test  software. 


The  Language  Implementation  rrKxiule  provides  an  area  to  discuss  features  unique  to  the 
specific  implementation  chosen.  Originally,  die  goal  was  to  abstract  out  the  underlying  language 
inq)lenientati(Mi.  Since  this  was  cost  jHohibitive,  it  provided  an  area  to  discuss  the  language 
specific  decisions  that  might  affect  program  port^iUty. 

Hguie  2.4.3- 1  illustrates  the  composition  of  the  Software  Decision-Hiding  Module. 


Figure  2.4.3-1  Software  Decision-Hiding  Module  and  Submodules 


-7- 


3.0  Data  Analysis 
3.1  General 


TRALAB  change  data  con^rises  developmental  integration  and  test  (l&T)  data  and  post- 
deployment  software  support  (PDSS)  dara  collected  during  two  time  periods.  The  first  period  was 
from  August  19S8  to  Api^  1989,  and  encompassed  data  on  software  design  and  code  problems 
found  during  Software  I&T  activity  and  System  I&T  activity  of  a  major  TRALAB  upgrade.  The 
upgrade  effort,  which  involved  several  thousand  source  lines  of  code  and  scenario/database 
updates,  prece^  initial  TRALAB  deployment  and  essentially  extended  the  initial  TRALAB 
development  effort.  All  problems  recorded  during  this  period  surfaced  during  the  execution  of 
customer-approved  test  plans  and  procedures,  targeted  at  verifying  required  system  behavior  as 
documented  in  the  Software  Requirements  Specification  and  the  System  Technical  Specification. 
As  such,  all  change  data  collect^  during  this  period  was  problem-oriented,  i.e.,  not  related  to 
system  enhancements  or  to  discrepancies  found  in  technics  documentation.  There  were  a  total  of 
230  identified  problems  and  proposed  changes  impacting  software,  scenarios,  and  databases  for 
this  “I&T  period”.  All  were  recorded  in  software  problem  reports  (SPRs),  reviewed,  and  resolved 
under  the  direction  of  a  contractor  Configuration  Control  Boaid  (CCB).  The  CCB  met  as  needed 
based  on  the  recent  number  and  severity  of  the  changes  and  problems  reported.  Fourteen  (6%)  of 
the  problems  were  attributed  to  user  error  (i.e.,  were  not  considered  system  problems)  and 
sunimarily  closed  with  the  resolution  that  no  modificaticMis  were  required.  The  remaining  216 
problems  required  rewoik  to  designs,  databases,  code,  or  scenario  scripts.  Although 
documentation  defects  were  not  recorded  with  SPR  forms,  the  effort  recorded  to  close  an  SPR 
included  the  updates  to  related  technical  and  user  documentation.  Two  hundred  and  one  of  the 
remaining  problems  (93%),  a  surprisingly  large  proportion,  were  directly  related  to  expet^ 
system  changes  that  were  listed  in  the  System  Technical  Specification  and  refined  during  initial 
design. 


The  second  time  period  was  from  August  1990  to  May  1991,  and  encompassed  data  on 
problems  uncovered  following  initial  deployment  of  the  system.  There  were  39  SPRs  generated 
during  this  *TDSS  period”.  Five  (13%)  of  these  were  attributed  to  user  error  and  summarily 
closed.  Four  of  the  remaining  problems  (10%)  were  unresolved  and  remain  open.  The  othn  30 
SPRs  resulted  in  updates.  Of  tlKse,  eighteen  (60%)  were  directly  related  to  expected  system 
changes. 

Unlike  the  I&T  SPRs,  PDSS  SPRs  included  enhancements  (i.e.,  12)  to  existing  TRALAB 
functionality.  Following  an  extensive  period  of  operational  use,  desired  TRALAB  functionality 
not  originally  specified  in  initial  Software  Requirement  Specification  was  captured  through  the 
configuration  management  (CM)  process. 

Figures  3.1-1  is  a  plot  of  the  accumulation  of  the  250  SPRs  that  required,  or  may  yet 
require,  TRALAB  modifications.  Figure  3.1-2  is  a  plot  of  the  associated  resolution/close-out 
activity.  Although  all  the  SPRs  were  handled  through  a  typical  multi-step  CCB  process.  Figure 
3.1-1  indicates  that  for  the  majority  of  the  I&T  SPRs,  there  were  no  significant  overall  delays 
between  SPR  submission  and  close-out  (i.e.,  the  final  step  during  which  the  CCB  verified 
complete  and  accurate  implementation  of  the  required  mo^fications). 

Appendix  A  contains  the  TRALAB  SPR  forms  used  to  document  change  data.  SPRs  could 
be  filled  out  by  i»oject  engineers  or  by  TRALAB  system  end  users  and  submitted  to  the  TRALAB 
Configuration  Manager.  The  Configuration  Manager  would  log  the  SPR  and  schedule  a  CCB 
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Figure  3.1*1  SPR  Accumulation 
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meeting  when  a  sufficient  number  of  SPRs  were  collected  or  when  the  severity  of  a  specific  SPR 
required  immediate  attention. 


The  CCB  would  validate  the  information  on  the  SPR  and  determine  a  course  of  action.  In 
most  cases,  a  software  engineer  would  be  assigned  the  responsibility  for  identifying  and  making 
the  necessary  modifications  to  the  software,  databases,  scenarios,  and  documentation,  and 
performing  regression  testing.  Following  successful  regression  testing,  the  Software  Modification 
Transmittal  (SMT)  form  wodd  be  completed,  logged,  and  submitted  to  the  CCB  for  review  and 
SPR  close-out. 

The  SMT  form  was  tailored  to  collect  the  information  necessary  in  determining  the  utility  of 
the  SCR-based  design  decomposition  t^proach  and  resulting  module  structure.  For  example: 

•  The  T\TE  OF  CHANGE  portion  of  the  form  contains  entries  that  would  indicate 
where  modifications  were  necessary  (i.e.,  in  SOFTWARE,  SCENARIO, 
DATABASE,  DOCUMENTATION,  and  OTHER).  TRALAB  software  designers 
had  tried  to  encsyisulate  several  of  the  most  likely  expected  changes  as  database  or 
scenario  entries.  Dat^ase  editing  and  authoring  tools  were  generated  to  facilitate 
database  and  scenario  updates.  There  was  interest  if  this  extra  design  effort  paid 
off. 

•  Number  of  software  modules  impacted  by  a  change,  and  the  lines  of  code  required. 
Since  inodiftcations  to  scenarios  or  datab^  files  woe  made  with  utilities  and  did 
not  require  recompilation  of  code,  such  changes  were  designated  as  zero-module 
impacts. 

•  The  EFFORT  field  included  the  engineering  staff  hours  necessary  to  complete 
updates  to  the  software,  scenario  and/or  dat^ase  files  and  to  modify  related 
technical  and  user  documentation. 

•  The  LABOR  GRADE  field  allowed  costs  associated  with  SPR  resolution  to  be 
calculated. 

Not  all  the  elements  of  the  SMT  proved  useful.  It  proved  difficult  for  project  persotmel  to 
complete  information  on  FAILURE  CAUSE  portion  of  the  form.  Accordingly,  Ae  following 
analyses  omit  the  subject  of  failure  causes. 

3.2  Analyses 

The  following  analyses  are  based  on  the  246  SPRs  that  resulted  in  updates  to  code, 
database  entries,  or  scenario  scripts.  The  I&T  tables  and  graphs  are  based  on  216  I&T  SPRs;  the 
PDSS  tables,  30  PDSS  SPRs. 

3.2.1  Areas  of  Change 

Table  3.2.1-1  is  a  summary  of  the  total  number  of  SPRs  that  requited  code  modifications  in 
e^h  of  the  major  TRALAB  functional  areas.  Any  SPR  that  impacted  tnore  than  one  functional 
area  has  been  counted  once  for  each  area.  SPRs  that  only  requi^  modifications  to  database 
records  or  scenario  script  files  are  considered  not  to  impact  software  and  are  not  counted. 


Table  3.2.1-1  SPRs  Impacting  Major  Software  Functional  Areas 

FUNCTIONAL  AREA 

NUMBER  OF  SPRs 

Hardware-Hiding 

10 

Behavior-Hiding 

105 

Software  Decision-Hiding 

87 

Twenty  nine  (12%)  of  the  SPRs  required  modifications  that  spanned  two  or  more 
functional  areas.  Seventeen  of  these  are  I&T  SPRs;  twelve,  PDSS  SPRs.  For  both  the  I&T  and 
PDSS  SPRs,  most  are  changes  that  rippled  across  both  the  Behavior-Hiding  submodule 
Simulation  Activity  (SMA)  and  the  Software-Decision  submodule  Simulation  Delivery  (SMD) . 
The  SMA  module  hides  the  behavior  of  the  collection  and  processing  systems  that  are  to  be  trained, 
while  the  SMD  module  hides  details  of  the  simulation  delivery  mecl^sm.  Unfortunately, 
changes  to  the  behavior  of  target  systems  generally  required  rnodifications  to  the  simulation 
delivery  mechanism  (SMD)  as  well  as  to  ^  simulations  of  target  system  behavior  (SMA). 
Fortunately,  modifications  to  the  SMD  module  fiequently  were  minor  and  related  more  to  the 
control  mechanisms  necessary  to  invoke  die  proper  simulation  activity  function.  A  few  of  the  29 
SPRs  required  modifications  to  the  i^Is  of  (^-hiding  modules  in  addition  to  modifications  to 
data  structures  used  in  module  implementations.  Changes  to  a  module’s  APIs  typically  require 
modifications  to  programs  in  other  modules  that  use  thid  API. 

Table  3.2.1-2  is  a  summary  of  the  total  number  of  SPRs  that  required  software 
modifications  in  each  TRALAB  third-level  software  module.  Any  SPR  that  impacted  more  than 
one  module  has  been  counted  once  for  each  iriqiacted  module. 

There  are  no  SPRs  that  required  modifications  spanning  all  three  functional  areas. 

3.2.2  Change  Confinement 

The  TRALAB  module  stracture  was  buUt  around  the  following  list  of  areas  of  expected 
change,  which  were  identified  first  in  the  System  Technical  Specification  produced  early  in  initial 
TRALAB  development 

-  terminal  interface  (e.g.,  blinking  approach,  bolding  ^proach,  window  addressing 
scheme,  keyboard  drivers) 

-  underlying  operating  system 

-  networking  environment  (i.e.,  Ethernet  protocols  to  establish  a  circuit  and  to  send  and 
receive  messages) 

-  simulation  messages  and  displays  (e.g.,  target-system  menus,  queries,  and  prompts) 

-  simulation  timing 

-  simulation  commands 

-  student  evaluation  criteria  and  reports 

-  student  monitoring  formats 

-  authoring  exchange  necessary  to  create/modify  scenarios 
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Table  3.2.1*2  SPRs  Impacting  Third*Level  Modules 


FUNCTIONAL  AREA 


RDWARE-HIDING 


VIRTUAL  OPERATING  SYSTEM 
VIRTUAL  NETWORK 
VIRTUAL  TERMINAL 


EHAVIOR-HIDING 


SCENARIOMAINTENANCE 
DATA  BASE  EDITOR 
SYSTEM  ADMINISTRATION 
SIMULATION  ACnVITY 
TRAINING  EVALUATION 
STUDENT  OBSERVATION 
MENU  PROCESSING 
CONTROL 


OFTWARE  DECISION-HIDING 


SIMULATION  DELIVERY 

VIRTUAL  DATABASE 

SYSTEM  DATA 

TARGET  DATA 

SYSTEM  ADMINISTRATION 
DATA 


NUMBER  OF  SPRs 


OPERATIONAL  DATA 
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-  specifications  for  key  data  structures  (e.g.,  scenario  data  formats,  target  system  display 
formats,  audit  trail  formats) 

-  access  policies  for  key  data  structures 

-  run-time  environment  (e.g.,  number  of  terminals,  number  of  student  errors  before 
instructors  are  alerted,  standard  duration  for  maintaining  student  audit  trails) 

-  lan^ge  implementation 

-  additional  authors,  students,  and  instructors 

-  additional  classroom  management  tools  (e.g.,  student  roster  generation  support,  student 
performance  trend  reports) 

The  original  TRALAB  designers  tried  to  minimize  the  potential  ripple  effect  of  carrying  out  such 
changes.  Table  3.2.2- 1  lists  the  proportion  of  SPRs  that  impacted  0, 1, 2,  and  3  or  more 
TRALAB  “lowest-level”  modules  (i.e.,  third  or  fourth-level  modules  with  no  submodules).  A 
large  percentage  (82%)  of  the  TRALAB  SPRs  impacted  zero  or  one  lowest-level  modules  (an  SPR 
resulting  only  in  riKxIifications  to  database  records  or  scenario  script  files  was  considered  to  impact 
zero  modules).  Only  3%  impacted  three  or  more  modules. 

Tables  3.2.2-2  and  3.2.2-3  break  down  the  data  for  Table  3.2.2- 1  in  terms  of  I&T  SPRs 
and  PDSS  SPRs.  No  PDSS  SPRs  impacted  zero  modules.  This  is  not  surprising  because  by  that 
time  the  target  systems  to  be  trained  were  con:4)lete  and  stable,  which  was  not  the  case  during  initial 
TRALAB  development.  There  is  a  much  larger  percentage  of  PDSS  SPRs  that  impacted  two 
modules,  but  none  resulted  in  modifications  that  rippled  across  three  or  more  modules. 


Table  3^.2<1  SPRs  and  Modules  Requiring  Modification 

PERCENT  OF  SPRs 

NUMBER  OF  MODULES  IMPACTED 

29 

0 

53 

1 

15 

2 

3 

3+ 

Table  3.2J2-2  I&T  SPRs  and  Modules  Requiring  Modification 

PERCENT  OF  SPRs 

NUMBER  OF  MODULES  IMPACTED 

33 

0 

53 

1 

11 

2 

3 

3+ 
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Table  3.2.2-3  PDSS  SPRs  and 

Modules  Requiring  Modification 

PERCENT  OF  SPRs 

NUMBER  OF  MODULES  IMPACTED 

0 

0 

59 

1 

41 

2 

0 

3+ 

Figure  3.2.2-1  presents  the  same  data  used  for  table  3.2.2- 1  over  time.  After  November 
1988  during  die  I&T  period,  the  proportions  of  SPRs  impacting  0, 1,  and  two  or  more  lowest- 
level  modules  remain  relatively  stable.  After  February  1^1  during  PDSS,  there  is  modest  overall 
growth  in  the  percentage  of  SPRs  for  which  modifications  ripple  across  two  or  more  modules. 


Figure  3.2.2-1  SPRs  Categorized  By  Number  of  Modules  Changed 
3.2.3  Ease  of  Change 

Table  3.2.3-1  shows  the  relationship  between  SPRs  and  ranges  of  required  resolution 
hours.  A  large  percentage  (82%)  of  TRALAB  problem  reports  required  a  st^  day  or  less  (i.e.,  8 
staff  hours  or  less)  to  complete.  Table  3.2.3-2  shows  an  even  larger  percentage  (90%)  of  I&T 
problem  reports  were  resolved  in  less  than  a  staff  day.  Table  3.2.3-3  shows  a  much  different 
situation  for  PDSS  SPRs.  PDSS  SPRs  seem  to  require  more  effort  to  resolve  on  the  average;  only 
23%  were  resolved  in  a  staff  day  or  less.  One  possible  reason  for  this  is  that  several  we^^  utility 
enhancements,  requiring  several  hundred  NCLSLOC  to  complete.  Examples  of  utility 
enhancements  include  utilities  to  simplify  databa%  generation  and  to  verify  database  integrity 
following  TRALAB  upgrades. 
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Table  3.2J>1  SPR  resolution  hours:  all  SPRs 

RANGE  OF  HOURS 

CUMULATIVE  PERCENT  OF  SPRs 

[0,  1] 

41 

[0,2] 

56 

[0,3] 

62 

_ [OJ] _ 

82 

Table  3.2.3*2  SPR  resolution  hours:  I&T  SPRs 

RANGE  OF  HOURS 

CUMULATIVE  PERCENT  OF  SPRs 

[0,1] 

46 

[0,2] 

63 

[0,3] 

70 

[0,8] 

90 

Table  3J23-3  SPR  resolution  hours:  PDSS  SPRs 


RANGE  OF  HOURS 


[0,1] 
[0,2] 
[0,3] 
[0,8 


imu 


ATIVE  PERCENT  OF  SPRs 


0 

0 

6 

23 


Tables  3.2.3-4,  3.2,3-5,  and  3.2.3-6  present  the  same  data  restricted  to  SPRs  that  were 
related  to  expected  tyj^  of  chwges  (total  is  219).  Generally,  a  greater  j^rcentage  of  SPRs 
involving  cl^ges  thitt  were  anticipate  by  initial  TRALAB  software  designers  were  easier  than 
unanticipated  changes. 
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Table  3.2.3*4  SPR  resolution  hours:  All  expected  change  SPRs 

RANGE  OF  HOURS 

CUMULATIVE  PERCENT  OF  SPRs 

[0.1] 

45 

[0.2] 

61 

[0.3] 

68 

_ \M1 _ 

90 

Table  3.2J-5  SPR  resolution  hours:  I&T  expected  change  SPRs 

RANGE  OF  HOURS 

CUMULATIVE  PERCENT  OF  SPRs 

[0.1] 

49 

[0.2] 

•  67 

[0.3] 

74 

_ [0i8] _ 

96 

Table  3.2.3<’6  SPR  resolution  hours:  PDSS  expected  change  SPRs 

RANGE  OF  HOURS 

CUMULATIVE  PERCENT  OF  SPRs 

[0.  1] 

0 

[0.2] 

0 

[0.3] 

6 

[0.8] 

27 

Hgures  3.2.3- 1  and  3.2.3-2  present  SPR  resolution  effort  over  time.  By  December  1989 
the  summary  patterns  found  in  Tables  3.2.3-1  through  Table  3.2.3-6  were  fairly  well  established. 
During  the  January  tt>  May  1992  time  frame  the  effects  of  the  PDSS  utility-rniented  enhancements 
increased  the  percentage  of  SPRs  requiring  more  than  one  day  to  coii^lete. 
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Figure  3.2.3*1  SPR  Resolution  E^ort  Categories 


I  HOtmOWLItt  —  HOim<...«P»T  >S^ 

Figure  3J23-2  SPR  Resolution  Effort  Categories:  Expected  Changes  Only 
3.2.4  Module  Size  and  SPR  Occurrences 

During  initial  TRALAB  development,  contractor  management  and  a  good  number  of  the 
development  engineers  were  {q)prehensive  over  what  appeared  to  be  some  rad^r  large  nnodules 
(see  T^le  2.3-1).  The  concern  was  tl^  such  large  entities  would  be  defect  prone.  Table  3.2.4-1 
shows  the  number  of  SPRs  per  1000  NCLSLOC  for  third-level  modules  in  different  size 
categories  (while  not  all  SPRs  involve  defects,  the  vast  majority  do).  The  data  con^ares  favorably 
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to  typical  industry  data  points  [JONES  81]  of  10-50  defects  per  1000  NCLSLOC  (average  for 
Americari-produced  code).  And,  as  of  the  middle  of  1991,  there  is  no  clear  relationship  between 
module  size  and  error  proneness.  Card  [85]  and  Conte  [86]  have  observed  similar  results. 


Table  3.2.4-1  Third-level  modules  and  SPR  rate:  All  SPRs 

MODULE  NCLSLOC 

COUNT 

SPRs  per  1000  NCLSLOC 

[0,  999] 

9.3 

[1000,  1999] 

3.8 

[2000,  2999] 

2.8 

[3000,  3999] 

8.3 

[4000,  4999] 

1.7 

[5000+] 

3.7 

We  do  not  have  an  answer  as  to  why  in  general  larger  modules  do  not  seem  more  error 
prone.  One  reason  may  be  tfiat  they  were  coded  with  more  care  and  experienced  more  scrutiny 
during  the  design  reviews  because  of  their  size.  A  more  pessimistic  explanation  may  be  that, 
because  test  covers  tools  were  not  used  to  support  module-level  testing,  there  may  be  numerous 
undetected  defects  within  the  larger  modules  due  to  the  scope  of  the  testing  required  to  fully 
exercise  all  module  paths.  TRALAB  module  testing,  however,  was  done  with  some  care.  For 
example,  black-box  testing  software  was  written  and  used  to  stimulate  the  external  interfaces  to 
TRALAB  third  and  fourth-level  modules  and  connate  actual  with  specified  results. 

We  have  a  likely  explanation  for  the  small  SPR/NCLSLCX^  value  for  TRALAB  modules  in 
the  NCLSLOC  size  catego^  [4000, 4999].  Actually,  there  is  just  one  such  module,  the  TR/JAB 
Menu  Processing  (TMP)  module,  ^rvices  provided  by  TMP  are  used  extensively  in  the 
inqrlementations  of  other  Behavior-Hiding  i^ules;  thk  is,  TMP  services  are  basic  TRALAB 
services.  To  avoid  troublesome  delays  during  I&T  typically  caused  by  flawed  basic-service 
software,  ThO’  services  were  tested  extensively  at  the  module  level  using  black-box  techniques 
before  the  module  was  baselined  for  actual  use. 

Table  Zl.A-l  shows  the  average  NCLSLOC  count  change  resulting  from  SPR 
modifications  for  third-level  modules  in  different  size  categories.  As  of  the  middle  of  1991,  there 
is  no  clear  rekuionship  between  module  size  and  the  magnitude  of  SPR  impact  on  NCLSLCiC 
counts. 
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Table  3.2.4-2  Third-level  modules  and  SPR  NCLSLOC 
impacts:  All  SPRs 

MODULE  NCLSLOC  COUNT 

MAGNITUDE  OF  NCLSLOC  COUNT 
DELTAS /SPR 

[0,999] 

11.3 

[1000,  1999] 

9.4 

[2000, 2999] 

6.0 

[3000,  3999] 

17.1 

[4000, 4999] 

7.0 

[5000+] 

23.2 

3.2.5  Cumulative  Resolution  Effort 

Figure  3.2.5- 1  shows  the  cumulative  average  hours  to  resolve  SPRs.  There  are  separate 
plots  fw  ^  SPRs,  all  SPRs  related  to  the  expected  changes,  and  all  SPRs  related  to  expected 
changes  that  could  be  handled  as  database  or  scenario  updt^s  (i.e.,  zero-module  updates).  The 
gR4)te  tend  to  support  that  TRALAB  engineers  benefit  by  engineering  for  change,  esp^nally  upon 
entry  into  PDSS.  In  particular,  they  seem  to  benefit  measurably  firom  encapsulating  specific 
changes  as  datadnise  entries. 

An  important  question  is  whether  the  differences  seen  in  Figure  3.2.5- 1  are  significant 
statistically.  The  popv^on  yielded  greatly  unequal  sample  sizes  fOT  the  groups  in  question.  There 
are  a  very  small  nutnber  of  unexpect^  changes  (i.e.,  27)  in  comparison  to  expected  (219).  The 
number  of  expected  changes  that  are  handled  as  database  updates  (70)  is  somewhat  small  in 
coii^)arison  to  the  remaining  expected  changes  (149),  and  these  two  data  sets  exhibit  some 
different  characteristics  (e.g.,  spread).  Unfortunately,  no  statistical  arguments  can  be  made  at  this 
time. 

3.2.6  Conclusions 

The  goals  of  the  TRALAB  program  were  to  specify,  design,  and  implement  a  system  that 
would  promote  the  ability  to  control  the: 

-  tipple  effect  when  implementing  expected  system  changes,  and 

-  effort  required  to  inq)lement  expected  system  changes. 

To  achieve  this,  initial  TRALAB  design  engineers  identified  expected  changes  early  in  the 
requiremrats  specification  process  and  factored  them  e;q>licitly  into  the  identification  of  TRALAB 
modules  and  the  specification  of  abstract  interfaces  to  ttese  iiKxiules.  Analysis  of  the  TRALAB 
change  data  tp  date  continues  to  indicate  informally  that  such  an  engineering  approach  controls  tire 
.  ri|>ple  of  change  modifications  and  the  effort  reqinred  to  implement  changes.  Additionally, 
allying  infonn^on  hitting  concepts  in  the  defmtion  of  the  module  structure  without  regard  to 
^ultimate 'module  NCXSLOC  sizes  does  not  seem  to  negatively  infract  defect  densities. 
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V 


Figure  3.2.5*1  Cumulative  Average  Resolution  Effort:  By  Close-Out  Date 
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Appendix  A.  TRALAB  Data  Collection  Forms 


SOFTWARE  PROBLEM  REPORT 

l&T  NO.: 

ORIGINATOR/ORG.  _ 

EXT.: 

SHEET _ OF 

DATE: 

PROBLEM  AREA; 

SOFTWARE 

HARDWARE 

DOCUMENTATION 

INTERFACE 

OTHER 

PROBLEM  WITH: 

ROUTINE 

DATABASE 

.  DOCUMENT 

SCENARIO 

TEST  FILE 

OTHER 

PROBLEM  IDENTIFIED  DURING:  ANALYSIS _  DESIGN _  OTHER _ 

SOFTWARE  TEST _  SYSTEM  TEST 

IF  TEST:  THREAD/CATEGORY _ CASE/SEGMENT _ 

BRIEF  DESCRIPTION; 

DETAILED  DESCRIPTION: 

PROBABLE  CAUSE: 


IMPACT: 


RECOMMENDATIONS/REMARKS: 

MODULES  REQUIRED  FOR  FK: 

DOCUMENT  UPDATES  REQUIRED; 

SETD  APPROVAL:  _ DATE: 

**CM  USE  ONLY  •• 

DATE  LOGGED: _  PRIORITY: _ 

ACTION  ASSIGNEE: _ CMO: _ 

CCB  ACTION: _ DATE: _ 

RESOLUTION  DATE: _ SMT  NO. _ DOCS.UPDATED _ 

Figure  A-1.  Software  Problem  Report 
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SOFTWARE  MODinCATION 

TRANSMITTAL  SPR  NO.: 


SMT  NO.: _ 

DATE: _ 

SHEET _ OF 

l&T  NO.: _ 


ORIGINATOR/ORG.:_ 

EXT.; 

DATE: 

MODULE  NAME: 

BASELINE  ID: _ 

CLASSinCATION  OF  HLES: 

TYPE  OF  CHANGE  SOFTWARE 

SCENARIO 

DATABASE 

DOCUMENTATION 

OTHER 

SOFTWARE  AFFECTED: _ 

DIRECTORY/LOCATION: _ 

CHANGED/ADDED  NEW/LAST 
DELETED  REVISION 


SOURCE  FILE  NAME/BRIEF 
DESCRIPTION  OF  CHANGE 


DOCUMENTATION  AFFECTED: 
DOCUMENT  ACCESSION  #S 


DOCUMENT  TITLEA^OLUME/DATE 


FAILURE  CAUSE 


TOTAL 


LABOR 

CORRECTION  TIME  GRADE 


_ ( 1 )  REQUIREMENT  CHANGE 

_ (2)  MISINTERPRETATION  OF  REQUIREMENT 

_ (?)  CHANGE  TO  TARGET  SYSTEM 

_ (4)  MISINTERPRETATION  OF  DOCUMENTATION 

_ (5)  IMPROPER  DESIGN 

_ (6)  IMPROPER  IMPLEMENTATION  OF  DESIGN 

CHANGED  _ 

_ (7)  TIMING/SIZING 

CHANGED_ 

_ (8)  DATABASE  ERROR 

_ (9)  INTERFACE  ERROR 

_(10)  OTHER _ 


09-ENG 
10- AE 
12-SE 
14-PE 

TOTAL  NUMBER  OF: 
MODULES 

LOCATION 


Figure  A'2.  Software  Modification  Transmittai 
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SOFTWARE  MODIHCATION 

TRANSMITTAL  SPR  NO.: 


SMT  NO.: _ 

DATE: _ 

SHEET _ OF 

I&T  NO.: _ 


ADDITIONAL  COMMENTS: 


FOLLOWING  ITEMS  UPDATED/DELIVERED: 


SOURCE 
STS 
MIMPS 
MINTS 
USER  GUIDE 


MDF/TDF 
DATABASE 
TEST  PROCEDURE 
STREAMING  TAPE 
OTHER: _ 


••  CM/SETD  USE  ONLY  •* 

CCB  APPROVAL:  _ _  DATE: 

CCB  APPROVAL/RECORDED: _  DATE: _ 

STATUS  OF  SOFTWARE  CHANGES:  OPEN: _ 

CLOSED _ 

STATUS  OF  DOCUMENTATION  CHANGES:  OPEN: _ 

CLOSED _ 


INTEGRATION  AND  TEST:  ACCEPTED:  ___YES  NO 
I&T  COMMENTS: 


TEST  BASELINE: _ 

SETD  APPROVAL: _ 

DATE: _ 

MASTER  BASEUNE  UPDATED  BY: 

DATE" _ 

COMMENTS: 


Figure  A>2.  Software  Modification  Transmittal  (Concluded) 


-24- 


